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ABSTRACT 



This study compared the use of two developmental 



alternatives: a Web Editor (WE) in combination with a customized 
template/shell (Teaching Not Teaching, T-N-T) and a WE only, for development 
of a Web-based lesson by pre-service teachers. Six hypotheses were tested to 
find whether the WE and T-N-T alternative was more efficient, effective, and 
appealing than the WE only development alternative. The efficiency, 
effectiveness,, and appeal of developing a Web-based lesson was examined using 
these variables: lesson creation time ; perceived effort; inclusion of six 
specified instructional components; functionality of six specified technical 
components; teacher appeal; and likelihood of intended future use (dependent 
variables) and the lessons development alternatives WE and T-N-T or WE only 
(independent variable) . A quasi-experimental design and t-test analyses were 
employed. Pre- service teachers (N=103) were assigned to one of the two 
development alternatives. Instruments included researcher-developed 
evaluations, self-report tools and the Computer Attitude Scale. Dependent 
variables investigated in this study included: lesson creation time; 
perceived effort; inclusion of six specified instructional components; 
functionality of six technical components; teacher appeal; and rate of future 
use. Statistical analysis of six hypotheses conformed that developing 
Web-based lessons using WE and T-N-T was more effective than using WE only. 
The analysis also revealed that neither developmental alternative was 
superior in terms of efficiency or appeal. (Author) 
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Abstract 

This study compared the use of two developmental alternatives; a Web Editor (WE) in combination with a 
customized template/shell (Teaching Not Teching, T-N-T) and a Web Editor (WE) only, for development of a Web- 
based lesson by pre-service teachers. Six hypotheses were tested to find whether the WE + T-N-T alternative was 
more efficient, effective, and appealing than the WE only development alternative. The efficiency, effectiveness, and 
appeal of developing a Web-based lesson was examined using these variables; lesson creation time, perceived 
effort, inclusion of six specified instructional components, functionality of six specified technical components, 
teacher appeal and likelihood of intended future use (dependent variables) and the lessons development alternatives 
WE + T-N-T or WE only (independent variable). 

A quasi-experimental design and t-test analyses were employed. Pre-service teachers (N = 103) were 
assigned to one of the two development alternatives. Instruments included researcher-developed evaluations, self- 
report tools and the Computer Attitude Scale (CAS). Dependent variables investigated in this study include: lesson 
creation time, perceived effort, inclusion of six specified instructional components, functionality of six technical 
components, teacher appeal, and rate of future use. Statistical analysis of six hypotheses confirmed that developing 
Web-based lessons using WE + T-N-T was more effective than using WE only. The analysis also revealed that 
neither developmental alternative was superior in terms of efficiency or appeal. 



Purpose of the Study 

A report from the National Center for Educational Statistics (NCES) (1999) details that although 
educational technology is considered a means for transforming education, only 20% of teachers feel well prepared to 
integrate technology into classroom instruction. This study examines a tool for promoting technology integration ^ 
through pre-service teacher skill acquisition and application of Web-based lesson development when presented 
through two different alternatives (WE + T-N-T and WE only). This study investigated whether using a Web editor 
(WE) in conjunction with a custom created template/shell (T-N-T: Teaching Not Teching) is more efficient, 
effective, and appealing for pre-service teachers to use in the development of Web-based lessons than using a Web 
editor only. The WE used in this study was Microsoft FrontPage 2000. 

Each pre-service teacher created a Web-based lesson using one of two lesson development alternatives. The 
Web-based lessons were then evaluated using three criteria; efficiency, effectiveness, and appeal. Comparison of 16 
variables/sub-variables was made between lessons created with WE + T-N-T and WE only. 

The three criteria for usability was applied in the evaluation phase to determine if one Web-based lesson 
development alternative provides better support for pre-service teachers compared to another alternative. Teachers 
must have access to technological innovations that help them meet the needs of a diverse audience, get the job done 
quickly, and have the ability to integrate best practices into the interface design and lesson content. 
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Description of the Sample 

During the 1999-2000 Fall Semester, 161 pre-service teachers were enrolled in four sections of ET 347 
Educational Technology Applications for Elementary Teaching , one section of ET 348 Education al Technology 
Applications for Middle Grades Teaching , and three sections of ET 349 Educational Technology App lications for 
Secondary Teaching . All of these students were invited to participate in this study. Permission to participate was 
received from a total of 144 students. Each student completed the Demographics Questionnaire and the Consent 
Form during the first class session. Of the 144 students in the study, half were identified as the Control group while 
the remaining students were identified as the Treatment group. 

The groups were selected by nonrandom methods; however, the intact sections were randomly assigned to 
treatment groups. Two of the four sections of ET 347 (elementary pre-service teachers) were randomly assigned to 
the Control group and the remaining two sections were assigned to the Treatment group. Two groups of ET 349 
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(secondary pre-service teachers) were drawn and assigned to the Control and one remaining section of ET 349 and 
one section of ET 348 (middle grades pre-service teachers) were assigned to the Treatment group. 

Forty-one of the 144 students (28.47%) who had agreed to participate in the study either did not complete 
their Web-based lesson or other required instruments and subsequently were eliminated from the study. Thus, this 
study’s sample consisted of 103 students, that is, 64% of the Educational Technology students who initially had 
been invited to participate. Final group size for the Control group was 52 and for the Treatment group was 51. 

The intact class sizes ranged from 16 to 26 students. The classes were scheduled for 50 minutes and met on 
the following days and times; two Monday classes at 12:20 and 1:25 PM, one Tuesday class at 9:05 AM, three 
Wednesday classes at 8:00 AM, 12:20, and 1:25 PM, one Thursday class at 6:00 PM, and one Friday class at 10:10 
AM. 

Research Questions 

The research questions were centralized around the usability/evaluation criteria efficiency, effectiveness, 
and appeal as follows: 

Efficiency 

RQi: Are pre-service teachers able to develop a Web-based lesson in less time when they use WE + T-N-T than 

when they develop a Web-based lesson using WE only? 

RQ 2 : Will pre-service teachers perceive that less effort is required by using WE + T-N-T than when they develop 
Web-based lessons using WE only? 

Effectiveness 

RQ3: Are the six specified instructional components (lesson goal(s), objectives written in performance-based terms, 
student performance, student performance evaluation, student-to-teacher contact, and location cues in the site) 
present when pre-service teachers develop a Web-based lesson using WE + T-N-T and when using WE only? 

RQ4: If present, which of the six specified technical components (navigation, image presence, mailto links, 
interactive mechanism, audio and video) are functional when pre-service teachers develop a Web-based lesson using 
WE + T-N-T and when using WE only? 

Appeal 

RQ5: What degree of teacher appeal is elicited from pre-service teachers' interaction with the development process 
of a Web-based lesson using WE + T-N-T and from pre-service teachers who use WE only? 

RQ6: Do pre-service teachers intend to continue Web-based lesson development with the lesson development 
alternative they used in the study in their future instructional settings? 

Research Design 

This study is quasi-experimental (Smith &, Glass, 1987). That is, the independent variable is an introduced 
treatment, although there will not be total control over which participants receive which treatment. Participants were 
selected by nonrandom methods and then the intact sections were randomly assigned to treatment groups. This 
research paradigm may be represented as follows: 

Group I R Oi O 2 

WE only 



Group II R Oi X O 2 

WE + T-N-T 

R = Random assignment of treatment to intact classes. Consent Form, Demographic Questionnaire 
Oi = Instruction on how to use the Web editor 
X = Intervention treatment (T-N-T) 

O 2 = Work Time-Log, Effort Questionnaire, Instructional Components Evaluation Checklist, Technical 
Components Evaluation Checklist, Computer Attitude Scale (CAS), and Intended Future Use 
Questionnaire, Summary Data Sheet 
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Data Analysis 

Results regarding the evaluation of the Web-based lessons created with a Web editor combined with a 
template/shell structure from "Teaching-Not-Teching" (T-N-T) and those created using a Web editor (WE) only 
developed by pre-service teachers are presented in this section. The evaluation was based on the comparison 
criterion of efficiency, effectiveness and appeal. These variables were comprised of 16 sub-variables. 

The t-test was used to analyze the data and determine if there was a significant difference between the 
Control and Treatment groups as measured by the 16 identified sub-variables. This study used six independent t- 
tests; therefore the probability of one or more Type-I errors was greater than the alpha set for any single t-test. Due 
to the multiple t-test error rate occurring from six t-tests, an alpha level of .0167 was set for each test resulting in an 
overall experiment-wise alpha of not more than , 1 0. Given the exploratory nature of the study, this error range was 
judged appropriate. The research null hypotheses are presented with the related analyses and results. 

Ho].’ There is no statistically significant difference between length of time required to develop a Web-based lesson 
using WE + T-N-T compared to using WE only. 

Based on the results of the independent t-test, Hqi was retained, indicating there was no difference in the 
time required by pre-service teachers to create a Web-based lesson using WE + T-N-T than when using WE only (t 
= -0.9457, p = 0.1733). Data were compiled and analyzed from the Work Time-Log. The results are presented in 
Table 1. 



Tnhlp L Work Time-Lo2 


Group 


Mean Time (minutes) 


SD (minutes) 


df 


P 


Treatment 
(n = 5i) 


519.1764 


215.3836 


101 


0.1733 


Control 
(n = 52) 


476.5384 


241.1787 







Participants in the study used the Work Time-Log to record the number of minutes they spent designing 
and developing their Web-based lesson. The six specific tasks itemized on the Work Time-Log were: 1) Tinkering 
with the computer program, 2) Storyboarding, 3) Collecting images, audio, video, and other cool stuff, 4) Thinking 
about my Web-based lesson, 5) Collecting or creating lesson content, and 6) Developing Web-based lesson with the 
editor. Although there was not statistical significance between the total number of minutes participants from each 
group took to complete their lesson, participants in the Treatment group spent, on average, 43 more minutes than did 
the participants from the Control group. 

Ho 2^ There is no statistically significant difference between perceived required effort when using WE + T-N-T 
compared to when WE only is used by pre-service teachers to develop a Web-based lesson. 

Ho 2 was retained, suggesting no difference in the perceived effort by pre-service teachers when developing 
a Web-based lesson using WE + T-N-T than when using WE only (t = 1 .8673, p = 0.0324). Data for this comparison 
were compiled from the Effort Questionnaire. The results of this analysis are found in Table 2 



Group 


ivcu Lj/iuri 

Mean Effort 


SD 


df 


P 


Treatment 


7.0588 


1 .9226 


101 


0.0324 


(n = 5i) 










Control 
(n = 52) 


6.3750 


1.7928 







H03: There will be no statistically significant difference between the frequency of presence of the six specified 
instructional components (e.g., lesson goal(s), objectives written in performance-based terms, student performance 
activity, student performance evaluation, student-to-teacher contact, and location cues in the site) in a Web-based 
lesson when developed by pre-service teachers using WE + T-N-T than when developed using WE only. 
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Ho 3 was rejected, indicating a significant statistical difference between the frequency of presence of the six 
specified instructional components in a Web-based lesson developed by pre-service teachers utilizing WE + T-N-T 
than when developed using WE only (t = 18.5048, p = 0.0000). Data were derived from a scale that ranged from a 
low of zero points to a high of six points. The data indicated a statistical favor for the WE + T-N-T group. See Table 
3 to review the results. 



Tahle3. Instructional Components 


Group 


Mean Score 


SD 


df 


P 


Treatment 
(n = 5i) 


5.3333 


0.9933 


101 


0.0000* 


Control 
(n = 52) 


2.1153 


0.7586 







♦ = a<0.0167 



The six specific instructional components itemized on the Instructional Components Evaluation Checklist 
are as follows: 1) Lesson goal(s), 2) Objectives written in performance-based terms, 3) Student performance 
activity, 4) Student performance evaluation, 5) Student-to-teacher contact, and 6 ) User-location within the site cues. 
The Treatment group scored higher than the Control group on all six instructional components. Not one Web-based 
lesson developed by the participants in the Control group included Objectives written in performance-based terms, 
or Student performance activity, or Student performance evaluation. 

Some participants from the Control group and the Treatment group designed what resembled an activity 
and/or quiz in their Web-based lesson but did not receive points for these components on the Instructional 
Components Evaluation Checklist. Operational definitions of these three variables follow. 

Objectives written in performance -based terms: refers to a statement that describes what learners should be able to 
do when they have completed the lesson. "What learners 'do' must be observable so that the learners know that they 
have learned and what they have learned" (Smith & Ragan, 1993, p. 91). 

Student performance activity: refers to eliciting specific behavior from the student based on the performance 
objectives for the lesson. 

Student performance evaluation: refers to supplying the student with feedback on their performance based upon 
student performance activity. 

The protocol in awarding points for these three variables was to first look for objectives written in 
performance-based terms. The performance objectives are the foundation for the other two variables and if there 
were no performance objectives contained in the lesson there could be no points awarded for either "student 
performance activity" or "student performance evaluation." As stated a-priori in the definition of “student 
performance activity,” performance was based upon stated objectives and if objectives did not exist in the lesson no 
points were awarded for student performance activity. Furthermore, no points could be awarded for student 
performance evaluation, because as stated a-priori in the definition, “student performance evaluation is to be based 
upon student performance activity.” 

H 04 : There will be no statistically significant difference between the frequency of functionality when the 
six specified technical components (e.g., navigation, image presence, mailto links, interactive mechanism, audio and 
video) are developed by pre-service teachers utilizing WE + T-N-T than when developed using WE only. 

Ho 4 was rejected, indicating significant statistical difference between the frequency of functionality of the 
six specified technical components: 1) Navigation, 2) Image presence, 3) Mailto links, 4) Interactive mechanism, 5) 
Audio, and 6 ) Video in Web-based lessons when developed by pre-service teachers utilizing WE + T-N-T than 
when developed using WE only (t = 9.7680, p = 0.0000). Data are derived from a scale that ranged from a low of 
zero points to a high of six points. The statistical preference was in favor of the WE + T-N-T. 

T-N-T group. The results can be found in the following. 
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Table 4. Technical Components 
Group Mean Score 



SD df p 



Treatment 


3.4901 


0.6744 


101 


0.0000* 


(n=51) 










Control 

(n=52) 


2.0769 


0.7883 







♦ = a<0.0167 



The scores on the Technical Components Evaluation Checklist were based on functionality of the six 
specified technical components. Not one participant from either the Treatment or the Control group attempted to 
include component 5) Audio or 6) Video in their lesson. Of the four remaining technical components, the Treatment 
group scored higher on average in every component. 

Ho 5 : There will be no statistically significant difference in the degree of teacher appeal expressed by pre-service 
teachers using WE + T-N-T toward developing Web-based lessons than from pre-service teachers using WE only. 

Hq 5 was retained, indicating no statistically significant difference between the degree of positive teacher 
appeal from pre-service teachers using WE + T-N-T to develop Web-based lessons than from those using WE only 
(t = -0.6403, p = 0.2617). Data were compiled and analyzed from the Computer Attitude Scale (CAS). The four 
areas associated with the CAS (low anxiety, high confidence, liking, usefulness) revealed no statistically significant 
differences between the two groups. The results can be found in Table 5. 



Group 


Mean Degree of Appeal 


SD 


df 


P 


Treatment 


118.2450 


16.7980 


101 


0.2617 


1) 

3 










Control 

(n=52) 


120.4807 


18.5723 







Ho 6: There will be no statistically significant difference between the likelihood of pre-service teachers’ anticipation 
for developing Web-based lessons with either alternative in their future instructional settings. 

Hq 6 is retained, suggesting no statistically significant difference in the anticipated future instructional 
development of Web-based lessons by pre-service teachers using WE + T-N-T to develop Web-based lessons or 
those using WE only (t = -0.3946, p = 0.3470). Data were compiled and analyzed from the Intended Future Use 
Questionnaire. See the results displayed in Table 6. 



Group 


Mean Score 


SD 


df 


P 


Treatment 
(n = 51) 


8.4019 


1.9053 


94.3 


0.3470 


Control 

(n=52) 


8.5769 


2.5540 







Appeal of the design/development process was addressed through the statistical examination of Hqs and 
Hq 6- There was no statistical significance in the differences between the Control group and the Treatment group on 
Hqs or Hq6- 
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Talk-Aloud Interviews 

The post-hoc talk-aloud interviews were completed by telephone with students from both the Control and 
Treatment groups. Two students were selected from each group based on most and least time recorded on the Work 
Time-Log. Two students from each group were also selected based on the highest and lowest compiled scores of the 
remaining five instruments (Effort Questionnaire, Instructional Components Evaluation Checklist, Technical 
Components Evaluation Checklist, Computer Attitude Scale, Intended Future Use Questionnaire). In theory, eight 
talk-aloud interviews would be performed, however, one of the individuals identified from the Control group fit in 
two categories (least time spent as recorded on the Work Time-Log and high composite score on the remaining five 
instruments), and therefore only seven participants were interviewed. 

Data obtained from the anecdotal talk-aloud interviews revealed that one participant from the Treatment 
group had continued using the T-N-T template/shell after the study ended. He had recently completed his seventh 
lesson in only 20 minutes. His completion time for the initial use of T-N-T was 450 minutes. Although this finding 
is based upon a single data point, the substantial, reduction in development time needs to be explored further to 
determine if multiple use of the WE + T-N-T template and WE only results in similar outcomes. 



Limitations 

Due to several factors, the researcher or reader would be vulnerable in drawing direct conclusions from the 
results of this study. The following limitations of this study should be considered when attempting to generalize 
from the findings and/or to replicate the study. 

The primary limitation of this study was the failure of the Control group to develop Web-based lessons that 
included the lesson components as defined within this study. According to the PTEP methods course syllabus, 
lessons components include goals, objectives, activities and performance evaluation. The Control group did not 
include these lesson components in the Web sites they created. Without such instructional components, the nature of 
the sites created is much more informational rather than instructional/educational. Had the Control group been 
required to rework their lessons so that the lessons included instructional components, the results of the comparisons 
between the Control group and Treatment group in regard to the efficiency (time and effort) and appeal (low 
anxiety, high confidence, liking, usefulness and intended future use) may have been substantially different. 

Restricting this study to a university setting is the greatest strength while also being the greatest weakness. 
Template design and interaction of pre-service teachers may not reflect the views of students at other universities, 
smaller or larger colleges, and/or community colleges. The university setting, however, offers the opportunity to 
assess students’ perspective as they create and layout their lessons using the template format. Ability to generalize is 
largely limited due to the sample selection. 

A third limitation is that this study uses a self-report. Self-reports are susceptible to "error” through the 
difficulty of eliciting honest, accurate responses. In this study, the participants are not under pressure to produce in 
order to be paid; nor are they reporting on an issue that will influence their grade, although they may be under a 
certain amount of time constraint pressure because of their own organizational capabilities. The assignment will be 
graded "Pass" or "Fail" based upon criteria in the grading rubric; student input from the self-report is not part of the 
grading criteria. These factors still do not guarantee the truthfulness of the responses. However, they do alleviate the 
pressure and stress to answer with "teacher pleasing" responses. 

A final limitation of this study is control for experimenter contamination. The intervention, T-N-T Web- 
based lesson template/shell, was designed and developed by the researcher and unquestionably there is researcher 
bias. The instruction for using the Web editor and the instruction for using T-N-T will be presented by the 
researcher. Maintaining fidelity during lesson presentation is a limitation. To minimize this limitation, adherence to 
guidelines for each scripted lesson will be described in Chapter Three. Another aspect of experimenter 
contamination is that data collection and data analysis is conducted solely by the researcher and subject to researcher 
bias. To minimize this limitation, the participants complete four of the six instruments. Of the remaining two 
instruments one is scored based solely upon component presence and the other is scored based solely upon 
component functionality. 
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Description of the T-N-T Template/Shell 

Essentially, the lesson template/shell is a complete Web-based lesson minus the content. The template/shell 
exists as a Web-based resource/tool for the teacher. This template/shell is designed to support a pedagogically sound 
linear presentation of materials and activities while concomitantly providing a multiplicity of instructional 
scaffolding cues designed to guide development of the content. Below are screen captures of two pages from the 
template. 




Figure L Screen Capture of the Navigation Bar 

The navigation bar includes seven links (Goals, Objectives, Readings, Activity, Quiz, Glossary and 
Teacher). Note that "Goals" is white (inactive link) and the others are red (active links). White text is a visual cue 
that the user is working in the Goals section. The title "Lesson Goals" directly below the navigation bar is another 
indicator of location and page purpose. Scrolling down the Lesson Goals page reveals a few lines of direct 
instruction to the developer. 

Each page of the template (excluding Glossary and Quiz) includes such direct instruction and a brief 
rationale (theoretical foundation) for the suggested approach to content development within each part of the Web- 
based lesson. The "Glossary" and "Teacher" links are set apart within the navigation bar. This segregation is 
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intentional as glossary and teacher sections are provided as Tools for the teacher/designer and not as having the 
same instructional rationale/purpose as the other five components of the lesson. The one link located on the 
navigation bar that does not open a Web page is titled "Teacher." This link is an E-mail link to the instructor. 

The lesson template also includes a ready-to-use ten question multiple-choice quiz. The interactive Quiz 
is a JavaScript program that solicits the user to answer questions relevant to the lesson content and receive 
immediate feedback on their performance. 

When providing content for the Quiz section of the lesson, the teacher/designer is prompted to insert 
questions pertaining to the lesson and potential answers (correct and incorrect). The feedback section of the Quiz is a 
program designed to give students appropriate information about their performance. Note that in the following 
screen capture, answers to the various questions have been selected and the user has clicked on the 'Calculate 

Results" button. r • r • 

The student performance feedback section of the Quiz page provides the student two sets of information. 

First, it furnishes a text-based assessment of their performance and secondly it allows the student to see which 
questions they answer correctly and incorrectly. 



1 Lesson Quiz - Microsoft Internet Explorer 
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Figure 2. Screen Capture of Performance Feedback section of the Quiz Page 
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Conclusions 

This study examined the efficiency (time and effort), effectiveness (inclusion of specific instructional 
components and functionality of specific technical components), and appeal (anxiety, confidence, liking, usefulness 
and intended future use) of two lesson developmental alternatives (WE only and WE + T-N-T) by pre-service 
teachers. Statistical analysis of six hypotheses confirmed that developing Web-based lessons using WE + T-N-T was 
more effective than using WE only. The analysis also revealed that neither developmental alternative was superior in 
terms of efficiency or appeal. 

Using a tool such a T-N-T significantly reduces the requirements for technical expertise in the development 
of a Web-based lesson. T-N-T also provides scaffolding for the creation of pedagogically sound instructional 
environments by focusing attention of the designer on instructional strategies and methodologies. 
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